Papers with representation models
CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent work on Word-in-Context fine-tunes models to investigate lexical meaning but only compares occurrences of the same lemma, limiting the range of captured information. |
| Approach: | They propose an extension to Word-in-Context to include inter-words scenarios by using a dataset and several models on a data set. |
| Outcome: | The proposed models provide efficient multi-purpose representations of lexical meaning that reach best performances in the experiments. |
LITE: Intent-based Task Representation Learning Using Weak Supervision (2022.naacl-main)
Copied to clipboard
| Challenge: | To-do texts are often short and under-specified, which poses a challenge for current text representation models. |
| Approach: | They propose a neural multi-task learning framework that extracts representations of English to-do tasks with a multi-head attention mechanism on top of a pre-trained text encoder. |
| Outcome: | The proposed model outperforms baseline models on four downstream tasks and achieves error reduction of 38.7%. |
Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification (C18-1)
Copied to clipboard
Jianyu Zhao, Zhiqiang Zhan, Qichuan Yang, Yang Zhang, Changjian Hu, Zhensheng Li, Liuxin Zhang, Zhiqiang He
| Challenge: | Existing representation models for text classification learn little structure information or rely on pre-defined structures. |
| Approach: | They propose a sandwich neural network to learn local semantic and global structure representations without relying on parsers. |
| Outcome: | The proposed approach achieves competitive performance on several text classification tasks. |
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters (2020.coling-main)
Copied to clipboard
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, Jun’ichi Tsujii
| Challenge: | Pre-trained language representations from Transformers have become the most popular choice for building NLP systems. |
| Approach: | They propose a new variant of BERT that drops the wordpiece tokenization system altogether . they propose 'characterBERT' module to represent entire words by consulting their characters . |
| Outcome: | The proposed model improves performance on a variety of medical domain tasks while producing robust, word-level, and open-vocabulary representations. |
AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal representation is crucial for E-commerce tasks such as identical product retrieval. |
| Approach: | They propose an approach which leverages the generative power of Multimodal Large Language Models to extract key attributes from product images and text and enhances representation learning through a two-stage training framework. |
| Outcome: | The proposed model achieves state-of-the-art on multiple downstream retrieval tasks, validating the effectiveness of harnessing generative models to advance fine-grained representation learning. |
Text Representation Distillation via Information Bottleneck Principle (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have recently shown great success in text representation field, however, the high computational cost and high-dimensional representation of PLMs pose significant challenges for practical applications. |
| Approach: | They propose a Knowledge Distillation method that distills large models into smaller representation models to reduce performance degradation after distillation. |
| Outcome: | Empirical results on two main downstream applications of the proposed method show that it reduces the risk of over-fitting and maximizes the mutual information between the model and the input data. |